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(57) Abstract: The software according to the 
invention incorporates a glossary management 
tool (fig. 2) that makes it easy for each client 
to customize terminology (fig. 2) to the needs 
of a particular business. With this tool, termed 
a glossary manager (fig. 2), a company can 
customize a number of lealure names (fig. 
2) in the system to provide a more familiar 
context (fig. 2) tbr their users. A system 
administrator can also customize the manner in 
which "thumbnail" or "preview" images (fig. 2) 
are presented. The system performs clustering 
on search queries, and searches media records 
multi-modally, using two or more approaches 
such as image searching and text searching. An 
administrator can tune search parameters. Two 
or more streams of metadata may be aligned 
and correlated with a media file, lacilitating 
later searching. The system evaluates itself. It 
folds popularity information into rankings of 
search results. 
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Method and Appaiatus for Digital Media Management^ Retrieval, and Collaboration 

Background 

5 Creative designers in vaiious fields work with materials fiom many different sources when 
they develop an advertisement, film, brochm*e, or other finished product. These materials 
may have been created origiaaUy by the creative designer, but for a different project, and 
stored in a media Ubrary or arcMve for later reuse. They may have been created by 
colleagues, and stored in an area that allows different individuals to collaborate on materials. 

1.0 Or they may have been created by professional artists for hcensing or sale. These materials 
are often called rich media files, or assets. Rich media files may include, but are not limited 
to, video, photography, graphics, audio, mixed media files, logos, presentations, and text. 
These media files can exist in the widest possible range of formats. It is a ponderous task to 
manage such assets. Annotation mechanisms for such assets include the system described in 

15 U.S. Pat. No. 5,493,677, assigned to the same assignee as the assignee of the present 

invention. Systems to manage such assets iQclude that described in U.S. Pat. No. 6,012,068. 

It would be desirable to have a management system for digital media which streamlines the 
task of accounting for rights to use such media, including copyright rights. Systems relatiug 
20 to rights management include tliose described in U.S. Pat Nos. 4,337,483, 5,201,047, 
5,260,999, 5,263,158, 5,319,705, 5,438,508, 5,629,980, 5,765,152, and 5,553,143. 

Users in different businesses may use different terminology to refer to tlie various media 
management functions. For example, some may use the term libraiy, while otliers use 

25 archive. Some may use project workspace, while otliers use share or collaboration tool. In 
raiany systems, changiag terminology requires tedious programming effort which risks 
introducing errors into the softwaie. It woxild be desirable to have a management system that 
conveniently peimits non-technical users to customize such terminology on a per-system 
basis. U.S. Pat. No. 5,850,561 describes a glossary construction tool for creating glossary 

30 from text. 
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Different con5)anies using the software according to tlie invention may have different 
corporate culture, inaage, and system context within the company. It would be desirable to 
have a management system that conveniently peimits non-teclinical users to customize the 
software on a per-system basis with respect to such corporate concerns. 

In a typical asset management system, users browse through media ffle collections and view 
£«iiibnail images of files to decide which files they want to work with. Tliese thumbnails are 
browseahles, or small representations o,f the actual images, videos, or otlier media files in the 
system. A browseable is created by optimizing an image or video firame for online browsing, 
so a browseable has lower resolution and smaller dimensions than the original ffle. It is 
commonplace, however, to find that the resolution and dimensions aie not well suited to the 
company. It would be desiiable to have a management system that conveniently permits a 
system administrator to customize the software in this respect. 

Natural language processing (NLP) techniques are well known, including their use in 
infonnation retrieval applications (Strzalkowski, 1993), (Stizalkowski, Perez Carballo and 
Marinescu, 1995), (Evans and Zhai, 1996). Past systems have attempted to improve upon 
vocabulary management techniques, for example as described in U.S. Pat. Nos. 5,251,316 
and 6,125,236. Past approaches for searching multimedia include U.S. Pat. Nos. 6,243,713 
and 5,794,249. ■ 

Clustering is well known, for example in U.S. Pat. No. 5,317,507, 5,758,257, 5,675,819, 
5,778,362, and 5,875,446. See also Buckley, Chris, J. Walz, M. Mitra and C. Cardie, "Using 
Clustering and Super Concepts within SMART: TREC 6" 

(http://trec.nist.gov/pubs/trec6/t6_proceedings.html); Zamir, Oren^ O. Etzioni, Madani, and 
y-'x-x.., KDD-97, "Fast And Intuitive Qustering Of Web Documents;" and KoDer, Daphne, and 
Mehran Sahami, ML-97, "Hierarchically Qassifying Documents Using Very Few Words. " 
Rankings relating to relevance are discussed in U.S. Pat. No. 5,642,502. 

The evaluation of information retrieval systems became an essential part of the field in the 
early '90s, and was strongly advanced by the TREC evaluations designed at NIST beginning 
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in 1993. The TREC evaluation contains different tracks, but the tiacks aU share the followiag 
common features: 

• Tliey are designed to provide a compai'ative evaluation between different systems, 
usually provided by different paiticipants. 

• Tlie evaluation is done using strict test conditions that contain a set of queries, a 
collection of documents, and relevance judgements. 

• The evaluations use evaluation scores such as precision and recall that supposedly 
predict real users' satisfaction fi'om a system 

While these evaluations are indeed helpful in compaiiag tlie performance of different IR 
systems, they do not provide constant feedback on the peifoimance of a live IR system. The 
base performance of an IR system could be at fust evaluated using a standard measmement 
such as the one above, but as more media files are added to a system and users submit quei-ies 
in an xmcontroUed manner, it is hard to predict or estimate the perfomiance of the systemi In 
addition, if the system does not faU into the initial TRBC evaluation tracks, it is necessary to 
develop an independent test case ~ a very costly task. It is desirable to have a better self- 
evaluation system for such a digital asset manager. 

Known annotation-related systems are discussed in U.S. Pat. No. 5,600,775, 6,006,241, and 
5,938,724. 

Summary of the invention 

The software according to the invention incorporates a glossary management tool that makes 
it easy for each client to customize terminology to the needs of a particular business. With 
this tool, termed a glossary manager, a company can customize a number of feature names in 
tlie system to provide a more familiar context for their users. A system administrator can also 
customize tlie mamier in wliich "thumbnail" or ''preview" images are presented. The system 
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performs clustering on search queries, and searches media records multi-mo dally, using two 
or more approaches such as image seai cliing and text searcliing. An administrator can tune 
seai-ch parameters. Two or more streams of metadata may be aligned and correlated with a 
media file, facihtating later searcliing. The system evaluates itself. It folds popularity 
ioformation into rankings of search results. 

Description of the drawing 

Tlie invention will be described wiQi respect to a drawing in several figm*es, of wMch: 

Fig. 1 shows a typical rights management editor interaction; 

Fig. 2 shows a typical customization interaction screen; 

15 Fig. 3 shows a screen permitting customers to adjust seai'ch parameters; 

Fig. 4 shows a sample of a face-matching system according to the invention; 

Fig. 5 illustrates a viewer according to the invention with text annotation box. 

Detailed description 



20 



The inventive software manages media files, or assets, making it possible to store and retrieve 
them, to collaborate witli colleagues to edit and work with; them, and to deliver them, either 
25 for further processing or in finished form for use. 

The workflow of the inventive software is designed to provide flexible choices among 
featiu-es that support a creative user who works with rich media files. As soon as a user logs 
on to the software (an optional step), she can immediately see if any system-internal messages 
30 are waiting in her system email inbox, either sent from otlier users collaborating with her on 
media files, or sent automatically by the inventive system to notify her of various occurrences 
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that may pertain to her work, such as a project update. The user can read her messages, reply 
to a message, or compose a new message to send to another system user. A composed 
message may consist of text only, or it may have a system-internal file attached. 

5 The user may wish to work on a project, such as an advertising campaign. She enters the 
project work ai'ea, and creates a new project. TMs creates a viitual workspace in the 
inventive system that is dedicated to holding media files and all their related files, including 
edited versions, colleague comments, official approvals, and so on. She decides which users 
will be allowed to share access to the project space. She may upload files that she has already 
10 created into die project work area. From there, she can send a media Sle off to a colleague 
for comments, editing, or approval. 

If she decides to rework a previously created file for use in the new ad campaign, she can 
search for the file in the file storage area of the inventive systenx She enters search terms, 

15 generally either by typing or by speaking into an iaput device, and finds an appropriate media 
file. She then saves the file to her new project. She may decide, instead, to have the full, 
higli-resolution version of that file sent to her for a different project. In the inventive system, 
delivery options for high-resolution files include on-liue delivery via ftp, http, or special 
network; via CD delivery, or even via mail delivery of the file in analog format. Ordering and 

20 file delivery in the inventive system are subject to several conditions, which the software 
verifies and tracks. These conditions include whether the user has the proper I'ights and 
permissions, not only to view the file but also to use or publish it. If use is subject to a fee, 
the fee must be paid, whether by billing, credit card, or other means. If usage requires 
personal approval, the request is routed to the right person and approval is granted and ' 

25 tiacked, aU within the inventive softwaie system 

Rights management If a user determines that she needs to use a rights-restiicted media file, 
she must engage in a dialogue with the iaventive software about her planned use of the file. 
Conpanies can customize sets of questions that the software presents to users, depending on 
30 which file a user wants to order. The user may have to answer whether she intends to publish 
the file, where it will be pubUshed, to what size audience, at what location, on what date. The 
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answers deterirdne whether the file can be made available, and can also determine the price. 
For example, if she is showing ten seconds of a movie for a one-time training seminar in 
Canada for an audience of fifty people, her cost may be far less than making those same ten 
seconds of movie available on the home page of a major entertainment Web site. And 
5 making the entire movie available on the Web may perhaps not be peiinitted at any price. 

When users order files in the software according to the invention, the rights management 
process controls every aspect of the order. A company may allow its employees to use its 
media files for personal use, marketing use, or xq an intemal organization project. Or they 

10 may allow all three uses in different situations. They can ti^ack the user's use of the files, they 
can designate the licensiag type for the files and make sure a rights agreement is iucluded, 
and they can enforce legal or company policy restrictions that may apply. The rights 
management process helps clients establish all of these elements so they can apply them 
appropriately to their stored media files. 

15 The invention described here makes it easier for iudividual chents to customize a rights 

management setup so that the Softwai-e will make the right pricing and availability decisions 
for each projected use. Central to the system is a question-answering paradigm: an 
administrator or a manager decides what questions should be presented to each potential user 
of a file, and what set of multiple choice answers to supply. Tlie software maps the questions 

20 and answers to a rights and usage package, and, if apphcable, each package is assigned 
pricing. 

For exanqjle, an administrator may determine that a file's availability depends on: 
25 • What department is giving the presentation? 
Who is the audience? 

Will the image be included in any handouts? 

30 

Will the image be displayed on an overhead? 
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When is the presentation date? 

Does the work contain images of any sculpture, painting, drawing, or other art object 
that may be copyrighted? 

The questions, in turn, ai*e grouped appropriately for a company's paiticular file collections, 
and ai e assigned to either groups of files or individual files. Questions and their answers may 
sei-ve as fundamental criteria for whether a file is made available; they may serve as modifiers 
to an available file by raising or lowering pricing dependiQg on usage; or they may simply be 
infonnational questions diat do not play a role in decision-making, but allow companies to 
track file use and user needs. 

The rights management process. When ordering files, users answer questions that ask for 
iuformation about how they are planning to use the files they order. The questions a user 
answers ai^e the usage questions a company designates for each possible use. This set of 
usage questions is called a rights package. Creating a rights package is the first step in the 
rights management process. 

A rights package, or set of questions, is added to other rights packages, to make up a rights 
package group. This allows a company to make more than one rights package available for a 
group of files. A rights package group is added to a particular license type. The license type 
describes the possible ways a file can legally be used. A rights agreement is added to define 
details clearly about the user's rights. In the final step, the rights agreement is associated witli 
a paiticular hcense type through a control table. The control table also contains pricing and 
approval information for each rights package. 

The core piece of this process is the creation and grouping of questions that cover a 
company's business and legal interests for their stored media file usage. Part of this process 
is completed with what may be termed a rights package question role editor. A typical editor 
interaction is shown in Fig. 1. 
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Glossaiy Customization Tool and Usability Customization Tool The software according to 
the invention incorporates a glossaiy management tool that makes it easy for each client to 
customize teraiinology to the needs of a particular business. With diis tool, termed a glossary 
manager, a company can customize a niuBber of feature names iq the system to provide a 
more familiar context for their users. 

The system also includes a usability management tool that customizes the software interface 
'look and feel'' to more closely reflect a company's coiporate culture, image, and to reflect 
the system context witliiii the company. A pait of the system tenned a usability manager 
allows a company to change system backgroimd colors, fonts, and the embedded logo. Tliis 
tool also makes localization into a different language simpler. A typical customization 
interaction screen is shown in Fig. 2. 

User-configurable browseable generation. In a typical asset management system, users 
browse tliiough media fQe collections and view thumbnail images of files to decide which 
files tliey want to work with. These thmnbnafls are browseables, or small representations of 
the actual images, videos, or other media files in the system A browseable is created by 
optimizing an image or video frame for online browsiag, so a browseable has lower 
resolution and smaller dimensions than the original file. Tlie browseable (the proxy of the 
actual media file) is what the user sees in the system according to the invention, and 
browseable size is generally set by a system administrator for use thioughout the system 
Importantly, the system permits generating these lower-resolution proxy files in a way that 
can be configured by the customer. 

Natural language processing techniques. In the system according to the invention, the 
natural language processing component is layered. The system architecture supports a 
conventional search algoritlim, wliich operates as a plug-and-play support application. The 
algoritlim is then used in a word expansion process. The word expansion is applied to both a 
user's search query, and to the stored files' captions, that is, descriptive metadata identifying 
individual files. 
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When a user submits a seaich query, the query undergoes several types of NLP processing 
(detailed below), and the result of each element in the processing adds new query components 
(for example, synonyms) and/or weights (which emphasize or de-emphasize words depending 
on theii* value to the query). The resultiag, or processed, query contaios expanded temis and 
5 weighting information that can be passed to any search engine. This makes it possible to use 
multi-search agent (or "federated search") applications as well. 

To expand a user search query, once a user query is received, it is divided into individual 
tokens, wliich may consist of single words or multiwords. (Multiwords are words die search 

10 engine recognizes as one unit, instead of as separate words. With this feature, when a user 
searches for sea lions, the system does not bring up lions by the ocean.) For this process, a 
variation of conventional pattern matclnng is used. If a single \^^ord is recognized as 
matcliiug a word that is part of a stored multiword, the processor decides whether to treat the 
single word as pait of a multiword, based on the contents of the stored pattern and the iuput 

15 pattern. Stored patterns include not just hteral words, but also syntactic categories (such as 
adjective or non-verb), semantic categories (such as nationality or government entity), or 
exact matches. If the iaput matches the stored pattern information, then it is interpreted as a 
multiword rather than as independent words. 

20 Processing then determines whether the resulting word is a function word (closed-class) or 
content word (open-class). The processor ignores function words. For content words, the 
related concepts for each sense of the word are retrieved from the semantic net. If the root 
word is unknown, the word is treated as a keyword, requiring an exact match. 

25 The system then matches the expanded, weighted search query with stored media files in 

order to return search results to the user. Media files are identified by the metadata associated 
with tliem, e.g. in a caption. Text-based files can be identified by metadata associated with 
them, and also by their content. When users import media files into tlie system, the software 
provides a workflow to add captions to each file. The files are then uploaded into the 

30 database, and the file captions are expanded and the infomiation stored for later file 
searching. 
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To expand the file captions, the processor looks up each word in tlie caption or text. Words 
that may he related in the semantic net ai-e located based on stored links, and the looked-up 
word, along with any related words, are all displayed as the "expansion" of that word. The 
processor determines whether the cun^ent word or phrase corresponds to a proper name, a 
5 location, or something else. If it con'esponds to a name> a name expansion process is invoked 
that displays the name and related names such as nicknames and other vaiiants, based on a 
linked name file. If die cuixent word or plirase corresponds to a location, a location 
expansion process is invoked that accesses a gazetteer and displays the location and related 
locations based on linked location information found in the gazetteer and supporting files. If 

10 the cuiTcnt word or plii^ase is neither a name nor a location, it is expanded using the semantic 
net links and weights associated with those links. Strongly related concepts are given high 
weights, while more remotely related concepts receive lower weights, making them less exact 
matches. Thus, for a query on car, documents or metadata containing car and automobile are 
listed highest, followed by those with sedan, coupe, and convertible, and then by more 

15 remotely related concepts such as transmission, hood, and trunk. 

Once the expansion is complete, the expanded word or phi ase is stored in an index database, 
where it is available for use ia searching as described below. Processing then returns to 
expand the next word or phrase in the text or caption. 

20 

The system according to the invention handles certain words and phrases individually. 
Multiwords are matched as a whole unit, and names and locations aie identified and looked 
up in die separate name and location files. Next, noun phrases and other syntactic units are 
identified. A part-of-speech tagger uses linguistic and statistical information to tag the parts 
25 of speech for sections of the user query. Only words that match by part of speech are 

considered to match, and if two or more parts of speech are possible for a particular word, it 
is tagged with both. After tagging, word affixes (i.e. suffixes) are stripped from query words 
to obtain a word root, using conventional inflectional morphology. If a word in a query is not 
known, affixes aie stripped fiom the word one by one imtil a known word is found. 

30 

An intermediate query is then formulated to match against the file index database. Texts or 
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captions that match queries are then returned, ranked, and displayed to the user, with those 
that match best being displayed at the top of tlie list. Li an exemplary system, the seaiching is 
implemented by first building a B-tree of ID lists, one for each concept in the text database. 
The ID lists have an entry for each object whose text contains a reference to a given concept. 
5 An entry consists of an object ID and a weight. The object ID provides a unique identifier 
and is a positive iateger assigned when the object is indexed. The weight reflects the 
relevance of the concept to the object's text, and is also a positive integer. 

To add an object to an existiug index, the object ID and a weight are inserted into the ID list 
10 of every concept that is in any way lelevant to the text. For searching, die ID lists of every 
concept in the query are retrieved and combined as specified by the query. Since ID lists 
contain IDs with weights in sorted order, determining existence and relevance of a match is 
simultaneous and fast, using only a small number of processor iastmctions for each 
concept-object paii\ 

15 

Search Technologies. The system allows users to search for media files with many different 
types of search queries. For example, users may subnait search queries by speaking them, 
typing them, copying them, or drawing them. 

20 The process of locating a particular file in a laige arcliive is a special area for innovation 
witliin the inventive software. Files aie characterized iu several ways. First, they have an 
identifier, genei*a]ly similar to a filename, wMch is imique within the system and makes it 
possible to link up aU the objects related to a file. These can include the actual 
high-resolution asset, lower-resolution thumbnails or other proxies for browsing, and 

25 infomiation about the file, or metadata. Searcliing can be performed on the file identifier, or 
it can be performed on the metadata. In the case of metadata searching, it is desirable to offer 
search altematives that go beyond the exact matching process involved in a standard keyword 
seaich. 

30 Some systems use conti'oUed vocabulary seai'cliing as an optimization of keyword searcliing. 
Keyword searches simply match exactly on any word in the user's search query that appears 

11 
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in the search target. (In the system according to the invention, the search target is the 
metadata describing a media file.) The set of potential keywords is quite lai^ge (as large as the 
vocabulary of English, or whatever language(s) are being used). If there ai'e no limitations on 
the seai'ch vocabulary that can be employed, a user can enter a search for puma and fail to 
5 find any files captioned as mountain lion or cougai , even though they aU refer to the same 
thing. Conti-oUed vocabulaiy is an attempt to address this problem, albeit at considerable 
cost. In a controlled vocabulary retrieval system, cataloguers aU agree to use the same terms. 
In practical terms, tins implies that, when cataloguing, they must check theii* controlled 
vocabulary lists and be sure not to deviate. Sometimes tools can be built to aid in tliis 
10 process, depending on the size of the controlled vocabulary. Similarly, tools can also be 
provided to searchers to control theii* search requests. However, controlled vocabulary 
systems do not scale beyond a few thousand terms, since it is impractical to look up every 
word in English for every seai ch. For broader retrieval systems, for faster cataloguing, and 
fox simpler searching, a different approach, is superior. 

15 

In addition to standard keyword and Boolean searching, the system software incorporates 
additional advanced teclinology for locating stored files. Rather than limiting searching to a 
controlled vocabulary, the system softwai'e includes natural language search, which allows 
cataloguers and usei^s to enq)loy any words in English (or whatever natural language the 
20 retrieval system is using). 

Natural language search incorporates: 

a semantic network of concepts 
25 additional linguistic tecliniques, including: 

phrase matcliing 

derivational morphology, in lieu of stemming 
part of speech tagging 
name recognition 
30 ' location recognition 

12 
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User-Tunable Search Parameters. Hie system according to the invention provides a screen 
for customers to adjust seaich parameters, to reflect their company use of stored media file 
collections. Tins is shown in Fig. 3. While tlie parameters may themselves be well-known ia 
a searcloing system, what is emphasized here is that the user (or, more likely, an 
5 administi'ator) can be granted access to such fundamental decisions about seaich as: 

(a) how good a match has to be before it is displayed to tlie user, e.g. 50%, and 

(b) how "creative" the search should be, i.e. how much should the search teims be 
10 expanded to include more distant synonyms and related terms. 

It should be borne in mind that the system according to the invention can be carried out on an 
intemet, meaning an IP-based network, and ia pai ticular may be carried out on the Intemet, 
meaning the global IP-based network. 

15 

Multimodal Search. Cmxently, search methods focus on textual input. The current invention 
incorporates new search techniques, and combines them in novel ways. 

Image search is becoming useful in commercial applications. In the system according to the 
20 invention, user search input is provided in a new way. Users may wish to select an existiug 
image as example input, so that a search consists of "Give me more images like this." 
Perhaps even more useful is the ability to select part of an image, analogous with "Give me 
more like tMs part." In the system according to the invention, identifying the part may be 
done in either of two exemplary ways: 

25 

1. Touch screen: user touches the screen to identify the portion of the image tlaat 
feeds into the search. 

2. Markup, using pen or other screen drawing metaphor, including through the system 
30 media viewer, which is described in more detail below. 

13 
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In addition, seai'ch modalities can be combined. This novel approach to seai^ch is particiilai'ly 
applicable to multimedia. Examples of combiaed, or multimodal, searches, include: 

- voice and text 

5 - touch screen and text 

- drawing and voice 

- drawing and touch screen 

and so on. 

10 

Vocabulaiy Management Ideally, the semantic net of concepts is quite large and attempts to 
incorporate every word or tenii in English (or other language being used for cataloguing and 
searching). No matter how large the semantic net may be, there will be a periodic need to 
expand or edit it. New words appear ia English periodically, and, although many may be 

15 slang and tlierefore not particularly important in a business context, some will be real new 
words and will be important enough to include. For example, rollerblading and in-line 
skating are relatively new terms in English, and depicting those actions is useful in 
advertising. So the terms need to be added to the semantic net. Semantic net/vocabulary 
maintenance is generally a manual process, particularly where the user has an existiog media 

20 library with a thesamiis and vocabulary management process. Such maintenance can also be 
performed automatically. 

To maintain a vocabulary for an information retrieval application tliat accepts user queries in 
natural language, a user maintaining a semantic net wonld track search queries in a query log. 
25 From the query log, he would detennine which words are actually novel and are candidates to 
be added to the system vocabulary, by expanding the query log using morphology, and 
possibly a spell checker and name identifier. The remaining terms that were not matched are 
the basis of a list for adding terms to the vocabulary, 

30 Tools to manage vocabulary include: 

14 
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• A moi-phological analyzer. This tool strips off any endings and morphological 
alterations in a query to find the stem, and checks to see if the stem is in tlie cuixent 
vocabulaiy. If the stem is not, the user doing the maintenance migjit try: 

• A spell checker, Tliis tool uses the conventional algoritlims to see if the supposedly 
new word is actually a naisspelliag of a known word. If it is not a misspelling, the 
xiser might try: 

• A name identifier. This tool checks to see if the supposedly new word is in a name 
configuration, in that it follows a known fii'st name in the query. If it does, it is added 
to a candidate name database. If it is not, it is proposed as a possible new word to be 
added to the system's vocabulaiy. 

Searching Audio/Video by Timecode Correlation with Search Criteria. Video and audio files 
can be timecoded, or marked such that the software in wliich they run can locate a specific 
firame (for videos) or measure (for audio) at any time. Importantly, the system according to 
the invention permits searching time-based media, including video and audio files, by 
combining two search elements. The first is a standard search, including but not limited to 
natural language search. The second is a time indicator, such as a SMPTE (Society of Motion 
Pictuie and Television Engineers) standaid timecode. Face recognition is an additional 
technology that can be used in searching. Face recognition is a subset of the more general 
teclinology of object recognition, and indeed techniques described here may extend to 
additional technologies as weU. 

The cmxent state of the art in face recognition technology makes it possible to take a 
manually created, labeled library of faces, and match faces fi:om a video to that library. For 
example, a user might work with a news video and use a face recognition program to label 
Nelson Mandela in it. The output of the face recognition program would be a time-coded 
segment, with start and stop times, of when Nelson Mandela was on camera, with the label 
"Nelson Mandela" attached to the time codes. While face recognition cmxently does not 
acliieve 100% precision or recall, it can still be useful. For example, one known system 
offers a contract rights management capability for films that demands time-coded segments 
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With names attached, and assinnes that users wffl create those manuaUy, so that the correct 
contract restrictions for each film segment can be attached to the right time codes. Given a 
sman Hbrary of the actors in a film, it would be possible to do a fast, automated noatch-up of 
time codes and actors, even with imperfect face recognition technology. Selecting the coixect 
actor from forty publicity shots would be much simpler thau selecting from among thousands 
effaces. 

Inqjortautly, the system according to the invention carries out the automated creation of the 
face library. Requii-ed elements include time-coded metadata (for example, the voice 
recognition transcript of a video), and the abihty to find the names of people in text. Each 
time a face and a person's name appear at the same time code, that occuirence is a potential 
new entiy for the face library. A user may run the face-to-name matcher for thousands of 
horn s and sift out the recurring matches as the most likely. In this way. a reference Kbrary of 
faces is created, and new material can be catalogued automatically. 

The software according to the invention approaches this by using alignment techniques to 
match up two or more streams of metadata. Forexample, a broadcast news program may 
contain closed captioning for the hearing-impaiied. It may also contain a separate description 
of the news footage, probably created manuaHy by tlie news department. The system 
according to the invention uses alignment to match the description, which is not time-coded, 
with the closed captioning, which is time-coded. This process allows the system to add time 
codes to the non-time-coded stream The software thm uses that new. derived stream (i.e. the 
description with newly added time codes), and searches for proper names within it. At the 
same time, using face recognition algorithms on the video stream, the software finds faces. 
The system tries to match up the faces with the proper names that describe who they are. 
This matched set provides us with a rough cut of a face (or object) reference library. This is 
exemplified in Fig. 4. 

Face recognition can also be employed to manage the Ubrary or ai-chive of media ffles. Media 
fibraries are assembled over time, often from disparate sources, and may contain multiple 
copies of a single media file, either with the same metadata or with different metadata. 
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Duplicate detection is therefore an important element of library and archive management, and 
face recognition (and, more generally, image recognition) can be leveraged to provide that 
capability. More broadly, for video, scene detection technology can assist in the process of 
identifying duplicates so that they can be pm'ged from the library. 

5 

Clustering and Other Ways to Determine Stored File Usage. Clustering involves combining 
user search queries in such a way that the searches can be analyzed usefuUy to provide 
answers to business questions. Clustering has received considerable attention in document 
infoimation retiieval (IR) and more recently, in video IR as a means of refining retrieval 

10 results based on user preferences or profiles, and to characterize the marketplace. The prior 
art contains many examples of clustering applied in information retrieval systems, but they all 
apply to search results returned to users rather than search queries submitted by usexs. In the 
system according to the invention, we cluster search queries by topic. We then use that 
information to adjust the collections of stored files so that the file collections wiU better meet 

15 users' needs. 

This system characterizes the information needs of groups (and subgroups) of users, with 
respect to a collection of media files (e.g. images, videos, sound chps, text, multimedia 
objects). Some common groupings include: 

20 

• seai ch queries that brought back no files 

• search queries that brought back no files the user was interested in 

25 • search queries that lead to expressions of user interest or sales 

Tills system applies clustering technology to user-submitted search queries, and to the files 
retrieved in search results. It also includes: 

30 • machine leamiug as applied to the above. 

17 
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• Characterizing the infoimation needs over time, by user type, or by other factors. 

• methods for reporting iile coUection needs to interested parties (for example, media 
suppliers). Tlie system informs a supplier that pictures of earthquakes are selling 
briskly, or that users keep lookiog for videos of dance performances but cannot find 
any. 



• methods for adjusting file collections based on the results of the clustering analysis, 
above. 



• novel clustering techniques. These include using a semantic expansion (such as. the 
WordNet hierarcMcal thesaurus) and phrase identification (such as noun phrase, and 
name and location identification) as the basis for the clustering. 

Before user queries can be analyzed, they must be expanded to a "common denonmatoi-'. To 
expand the user search queries, we use natural language teclmiques. Specifically, we treat 
each query as if it were metadata within our system, as described in the NLP section, above. 
Each query is expanded thix)ugh the application of a semantic net, so that it contains 
synonyms and related temis. This expansion provides a set of features to which we can apply 
Standard clusteiing technology. 

Obtaining valuable information on user preferences for stored files begins with deciding what 
iirfonmtion a cKent wants to understand. The data set can be selected acbording to various 
criteria, including: 

• queries fiom a particular subset of users (e.g. registered users, users by industry, 
new users) 

• queries that lead to success (sale or oflier indication) 

• queries that lead to failure 
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A first step is to select the data set on which clvistering is to be performed. Li an information 
retrieval (IR) context, clustering can be perforaied on queries or on assets to be retrieved 
(documents, images, video, audio, mixed media). A sample query set may include short 
queries, as is standai*d on Web searches, long queries, as seen in TREC (U.S. 
5 government-sponsored text retrieval conferences), or as produced by QBE (query by 
example), in which an asset or set of assets are themselves inserted iato a query. 

A second step is to perfomi analysis on tlie queries usiag, for example, linguistic methods, 
such as: 

10 

• Tokenization: determine word/token boundaries. In English, tokenization mostly 
coincides with spaces between words, with certain complications (Alzheimer's = 1 
token vs. she's = 2 tokens). 

15 ' Morphology or stemmiug: removed tense and plural markers and other affixes to 

find the word root. 

• Identify names, locations, norm phreises: using a pattern matcher or other 
methodology, determine words were groupings for special handling. For example, for 

20 names, match certain kind of variance; for locations, match subset; for noun phrases, 

we to complete and head-only matches higher than modifier home-only matches. 

A tliii'd step is to expand the queries. Ideally, tliis step includes expansion using a thesaxjrus 
or semantic net of synonyms, super- terms, and o flier relationships. 

25 

A fourth step is, for each of flie terms in each expanded query, assign a weight based on how 
close that tennis to the original query. The exact weighting will vary by application, but the 
basic understanding is that more closely matching terms are weighted close to 100. 

30 A fifth step is to create a vector for each expanded query. In order to apply a statistical 
clustering algoritlim, we arrange die vectors into a matrix. 

19 
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A sixth step is to a pply a statistical clustering algorithm in orier to gK>up staflar qtKries 
together. A hierarchical or tear clustering strategy may be used, dq^nding on whether- ^ 
desned clusters are hierarchical ornot. Qustering may allow overlap, which means that a 
query may appear in more than one cluster. 

A seventl. step is to apply the clustering algorithm until the stopping condition is met e g 
the de^ed number of clusters is obtained, or a combination of cluster munber and desired 
distmctiveness of clusters is reached. 

An eightla step relates to the clusters. Clusters are most useful to a hiiman observer if they 
bea,- useful naz.es that reflect the content. Use a semantic net hierarchy, combined with term 
frequency in a reference corpus, to identify the lowest-level term in the hierarchy that 
subsumes all the queries. 

Foldins Popularity Into Rankings. Many information retrieval appHcations cun ently 
incorporate relevance feedback into their judgements of how to rank search results returned to 
a user, hx all cases, however, the past systems utilize explicit user feedback, not implicit 
feedback. That is. they rank files by requiring a user to indicate wloat items he is interested in 
once a set of items is returned by a search. Importantly, in the system according to the 
invention, the system discerns impKcit popularity rankings based on a ranked set of user 
actions. Thesystemflxenthenusesthoserankingstore-ranksubsequentsearchresults The 
user actions from which popularity may be detennined include, but are not limited to: 

• whether a file is placed in a projects folder or other work space 

• whether a file is placed in a shopping cart 

• whetlier a file is purchased 

m addition, implicit popularity rankings may be derived from one user or set of users and 
applied to a different user or set of users. For example, if User A places a media file in a 
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projects folder or shopping cart, tlie information on her activity caa also be used to re-raqk 
search results for User B, who is in some way similar to user A. In another example, if users 
with ".edu" e-mail addresses buy certain tilings, it makes sense to re-rank search results to 
favor those things when showing results to other ".edu" users. In the system according to the 
5 iuvention, if registered users who work for advertising agencies have placed certain items in 
theu' shopping carts, other advertising agency employees can have theii* search results 
re-ranked to favor those itemis. The opposite can be true as well: the same system can be used 
to disfavor certain items because they have been sold too many times to advertising agencies, 
for example. 

10 

Retrieval system self-evaluation. Hie retrieval system according to the invention does not 
answer any of the TREC tracks — a media file is described by a short pai'agraph or by 
keywords. Even though it is possible to develop a unique test collection for purposes of 
evaluation, it will not necessarily predict the perfomiance of new systems, or even for 
15 existing ones. 

What is now described is a method for ongoing evaluation of IR system performance based 
on search results combined with user feedback. This approach enables the system to alert a 
human system manager about observed degradation in the system performance. As the 
20 perfomiance evaluation system becomes more knowledgeable through methods of machine 
learning, the system is desirablyl able to change its own parameters in order to improve its 
perfoimance. 

'Performance" as used iu this inventive system is a score on a scale of user satisfaction with 
25 the respect to search results rankings. User satisfaction is determined by tracking the user 

actions together with tlie system's data. For example, for a very good performance, the user 
would submit a search query and purchase the media file that was ranked first. Possible 
indications of poor perfomiance might include a user issuing several consecutive queries, 
getting thousands of results, and terminating the session witliout fui ther examining any of the 
30 media files. 
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Viewer. The software appUcation according to the invention includes a viewer for 
amotations, or mark-ups, of Mi^otion digital stiUixnages, video frmnes, or otl,er indexed 
data structures, '^^e viewer, preferably accessible on a personal computer with a Web 
browser, serves as a helper appHcation for the underlying appHcation of the system according 
to the invention. Tl.e viewer aUows users to view ^d xnaxk up (conni^nt on) inedia on a 
sepai-ate, see-through, lightweight layer. The viewer supports annotations that axe created and 
renderedsott.attheyappeaz-overlaidonfl.erelatedxnedia Position, size, tae number and 
other attributes of annotations are saved in a separate annotation file, and when played back 
ttie annotations appear at the saved location and with the saved attributes. Preferably, to the 
gi-eatest extent possible, creating and editing of annotations is be done in WYSIWYG (what 
you see is what you get) fasliion. Tins is illustrated in Fig. 5. 

The annotations may consist of text, graphics, and digitized audio, and are stored separ ately 
from the original file informatioa An indexing scheme relates individual annotations to 
individual video frames. On full motion video playback, the annotations are displayed on the 
originally annotated frames. One central part of the system according to tlae invention is the 
use of a downloadable, view-only application, which enables users who do not have the full 
power to create annotations nonetheless to be able to view annotations created by others. 

M an exemplary embodiment, ffles are stored centrally. Preferably the viewer pn^vides 
security, e.g. SSL, server authentication, and proxy authenticatioa The underlying 

appHcationaccording to the invention powers the full lifecycleof digital media from content 
conaboration. deep storage, search and retrieval using a natural language search and the 
abmty effectively to move bandwidth-intensive media over the Web. The software is 
preferably Web-based. 

Tie vfewer k a standalone. plaHbn^nadve "helper" appHcation ,ha, is nomaBy hunched by 
a Web browser, froma specific link on a Web page that is par. of .he Web server of 
s,s.em according .o fte invention. Tl« viewer has fte abiHQ- » upload annomion da« back 
.0 d» Web server. Except for fl« two acions of lannching and uploading, to viewer does 
no. requ^e any ne.work connection with the Web server or fl» Web bmwser. 
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The current viewer in the system according to the invention on a PC or Macintosh platform 
allows the user to view and annotate various video, still image, audio, and text formats, 
includiag MPEG, QuickTime, or RealMedia videos, bitmap, jpeg, and tiff images, MP3, 
.wav, .aiff, .au, and .snd audio files, and .pdf document files. The viewer cmxently has niae 
modes in which it can operate, all of which would be relevant to the mobile user. They are: 

• annotate a media file 

• select a keyfi-ame 

• select storyboard firames (and optionally a keyfirame) 

• set the download (cache) dii*ectory 

• send a message and, optionally, attached files to anotiier user 

• download a media file 

• upload media files to a project 

' upload media files to a public collection 

• select a playhst 

The current (Mac and Windows) viewers can be launched in any of these nine modes, 
generally by receiving an XML annotation fiOle. 

Attributes common to all annotation types are: position, size, firame number (video only), and 
page number (multipage document only). AU of the grapliical attributes such as colors and 
fonts are modifiable using platform-native dialog boxes. These attributes can be set for new 
annotations as well as previously created ones. 

For video, each annotation is recorded with the timecode of the specific fiame on which it 
was created. When movies with annotations are played back, the viewer stops playback at 
each firame where an annotation exists. The viewer also contains a diop-do wn list of all 
annotated fi-ames (by timecode) so that individual annotated firames can be viewed quickly. 
Controls are also provided for stepping by a single fi-ame in either direction, as well as a 
control displaying the timecode of the current fi-ame where the viewer is paused. 
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The PDA (or handheld or wearable) version of the viewer builds on the same general 
architecture. The sei-ver generates a simple XML annotations file, sends it to the cKent (i e 
the viewer via the Web browser), and handles the retuxned HTTP messages from the viewer 
The foUowmg annotation types are desirably supported on a PDA version: 

■^yPe Attributes 

Freehand drawing (pen) Line widthColor 

Transparent text Font name, Font size, Font color 

Sticky note po^t uame, Font size. Font color, Background color 

S^"^'* Audio data 

la this way. the system according to the invention supports the fim life cycle of media- 
allowing organizations to expedite the process of searching, coflaborating and distributing of 
digxtal assets, such as imagery, video files, pictures and text documents. Users desirably have 
the freedom to coUaborate anywhere at anytime using tl.e annotation viewer according to the 
mventiononaPDA. Users are preferably able to annotate video and still ix^ges with a 
virtual pen, sticky note, text and even voice using a wireless modem Annotations are 
preferably capable of being added, edited, moved, or deleted. 

A PDA-based digital media mmiagement solution takes digital deUvery and coflaboration to 

thenextlevelbyeliminatingthaneedofatraditionalPC^dEthemetconnection. In 
addxtxon, users no longer need to cax.y laptops to stay up to date on work involving video or 
stfll images. 

Pnrffecably ae viewer suppom Ix-fl. HTW auteication and Proxy auftenticatic™. As an 
opto,,, to viewer diaplaya a login diajog where fte user can en.er a „ser ID and passworf u, 
*e used for auaendcation. Since 4. viewer is activated and Cosed irequenfly, i, should no. 
pronapt .he user to log in each tin,e it connects to the server. Tie viewer instead pt^vides a 
mechamsm for saving to user ED and passworf on to local worfetaaon (in encrypted fbnn) 
and passing this infonnation to to server witout any pronipting. It is desimble to. to 
viewer support SSL encryption for an of i.s HTTP reqnesB. 
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1 . A method for use with &st and second metadata streams with respect to a video stream, 
the & st metadata stieam time-coded with respect to the video stream and the second metadata 
stieam not time-coded with respect to the video stream, the method comprising the steps of: 

aligning the second metadata stream with the first metadata stream; 

adding time codes to the second metadata stream, based on the alignment; 

searching for proper names witliin the second metadata stream; 

finding faces within the video strean^ 

matcliing faces with proper names; and 

placing the matched faces and proper names into a reference library. 

2. The method of claim 1 wherein the video stream is news footage, wherein the first 
metadata stx-eam is closed captioning for the hearing impaii-ed, and wherein the second 
metadata stream is a separate description of the news footage. 

3. A method for use with a system storing myriad digital media files, the method comprising 
the steps of: 



generating myriad first thumbnail images, each first thumbnail image corresponding to one of 
the digital media files, each first thumbnail image having lower resolution and smaller 
dimensions then ihe corresponding digital media file, each first thumbnail image having first 
dimensions; 



displaying a subset of the fii st thumbnail images, the subset 



comprising more than one and 
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less than aU of the myriad first thumbnail images, the subset created as a result of input firom 
a fiist user; 

receiving a configuration command fi*om a second user, the configuiation command • 
5 indicative of second dimensions differing from the first dimensions; 

generating myriad second thimibnail images, each second diumbnail image correspondiag to 
one of the digital media files, each second thmnbhail image having lower resolution and 
smaller dimensions then the corresponding digital media file, each second thumbnail image 
10 having second dimensions; 

displaying a subset of the second thumbnail images, the subset comprising more than one and 
less than all of the myriad second thumbnail images, the subset created as a result of input 
from a third user. 

15 

4. The metlaod of claim 3 wherein the first and tliird users are the same user. 

5, The method of claim 3 wherein the first and third users are different users. 

20 6. A method for use with a system storing myriad digital media files, the method comprising 
the steps of: 

generating myriad fii'st thumbnail images, each first thmnbnail image corresponding to one of 
the digital media files, each first thumbnail image having lower resolution and smaller 
25 dimensions tlien the corresponding digital media file, each first tliumbnail image having first 
dimensions; 

displaying a subset of the first flimnbnail images, the subset comprising more than one and 
less than all of the myriad first thumbnail images, the subset created as a result of input from 
30 a first user; 

27 



wo 02/19147 

PCT/USOl/26841 

receiving a coijfiguration coimwd from a second user, the configuration command 
indicative of second resolution differing from the first resolution; 

generating myriad second thumbnail images, each second thumbnail image corresponding to 
one of tiie digital media files, each second thumbnail image having lower resolution and ^ 
smaner dimensions then the corresponding digital media file, each second thumbnail image 
having second resolution; 

displaying a subset of the second thmnbnail images, the subset comprising more than one and 
less than aU of the myriad second thumbnail images, the subset created as a result of input ' 
from a thii-d user. 

7. The method of claim 6 wherein the first and third users are the same user. 

8. The method of claim 6 wherein the first and third users are different users. 

9. A method for use with a system storing digital media records, the system comprising a 
vocabulary file of words keyed to the digital media records, the method con5,ming the steps 

of: 

receiving a seai ch queiy by computer from a user, the search queiy including words; 
logging die search query by computer, yielding a query log; 

processing the query log to identify words in the query log which are not words already 
included in the vocabulary file and which are not words which are variants of words already 
included in the vocabulary file, the processing performed by computer, said words defined as 
"not-found words;" 

adding the not-found words to the vocabulary file, and 
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10. The metliod of claim 9 wherein the vocabulary file further compiises a thesauius and the 
step of keying the not-found words to the digital media records comprises adding the not- 
foimd words to the tliesaurus. 

1 1. The method of claim 9 wherein the addiag and keying steps are perfomied manually. 

12. The method of claim 9 wherein the adding and keying steps are performed automatically. 

13. The method of claim 9 wherein the processing step comprises morphological analysis. 

14. The method of claim 9 wherein the processing step comprises spell checking. 

15. Tlie method of claim 9 wherein the processing step comprises name identification. . 

16. A metliod for use with a system storing digital media records, the system comprising a 
data file defining rights management questions and multiple-choice answers thereto, the 
method comprising the steps of: 

receiving, firom a first user, a selection of rights management questions and respective 
multiple-choice answers for the data file; 

receiving, from a second user non-identical to the first user, a query regarding a first digital 
media record; 

presenting, to the second user, a first rights management question among the selection of 
rights management questions; 

receiving, fiom the second user, a multiple-choice answer from among the respective 
multiple-choice answers with respect to the first rights management question, and storing 
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same; 



, ■ 3senting, to tlie second user, a second rights manageinent question ainong the selection of 
rights management questions; 



receiving, from the second user, a multiple-choice answer from among the respective 
multiple-choice answers withrespect to the second riglits management question, and storing 
same; 

presenting, to the second user, a third rights management question among the selection of 
rights management questions; 

receiving, from the second user, a multiple-choice answer from among the respective 
multiple-choice answers with respect to the third riglits management question, and storing 
same; 



deriving pricing infonnation for the second user with respect to the first digital media record 
from file stored multiple-choice answers; 

receiving, from a tlm d user non-identical to the first user and non-identical to the second user, 
a query regarding a second digital media record; 

presenting, to the third user, a fourth rights management question among the selection of 
rights management questions; 

receiving, from the third user, a multiple-choice answer from among the respective multiple- 
choice answers with respect to the fourth rights management question, and storing same; 

presenting, to the third user, a ftfth riglits management question among the selection of rights 
management questions; 
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receiving, from the third user, a multiple-choice answer from among the respective multiple- 
choice answers with respect to the fifth rights management question, and storing same; 

presenting, to the tliiid user, a sixth rights management question among the selection of rights 
5 management questions; 

receiving, fr om the third user, a multiple-choice answer fi'om among the respective multiple- 
choice answers with respect to the sixth rights management question, and storing same; 

10 deriving pricing information for the third user with respect to the second digital media record 
fi'om the stored multiple-choice answers. 

17. The method of claim 16 wherein: 

15 the first and fourth rights management questions are the same; 

the second and fifth rights management questions are the same; and 
the third and sixth rights management questions are the same. 

20 

18. A method for use with a system storing digital media records, the system comprising a 
search engiae the operation of which is defined by a plurality of stored parameters, tlie stored 
parameters defining, for the search engine, an extent to which the seaidi engine expands 
search terms, the system communicatively coupled to an internet and providing a hypertext 

25 transfer protocol server interface to the iatemet, the method conq)rising the steps of: 

receiving fiom a first user, via a hypertext transfer protocol client session, a change to one of 
the parameters; 

. 30 storing the change to the one of the stored parameters; 
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receiving from a second user, via a hypertext transfer protocol client session, a search request; 
passing the search request to the seaich engine; 

perfoi-ming. by the search engine, a search detem±aed by the searchrequest and detennined 
by thv. stored parameters, and deriving search results therefrom; 



reportiag the seaich results to the second 



user. 



19. ^ anethod for use with a system storing digital media records, the system comprising a 
seaicu engine seax-ching said digital media records, the method comprising the steps of: 

displaying all or part of a &st digital media record to a user; 

receiving, from the user, first infonnation indicative of a selection of less than aU of the 
displayed portion of the first digital media record; 

receiving, from tlie user, second information indicative of text; 

performing, by the search engine, a se^ch of the digital media records with respect to the first 
mfom^tion and the second information, and deriving search results therefron^ and 



displaying the search results to the 



user. 



20. The meflaod of claim 19 wherein the step of receiving first infoi^ation comprises the 
user touching a screen to identify the portion of an image displayed on the screen 

21. The mefliod of claim 19 wherein the step of receiving first information comprises the 
user drawing on a pad to identify the portion of an image displayed on the screen 



22. Tlie method of claim 19 wherein the step of receiving second information 



comprises the 
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user typing at a keyboaid. 

. 23. The method of claim 19 wherein the step of receiving second infomiation comprises 
receiving' voice information and perfonxiing speech recognition upon the voice inforaiation. 

24. A method for use with a system storing digital media records and comprising a search 
engine searcliing said stored digital media records, the method comprising the steps of: 

receiving seaich requests from users; 

logging the search requests; 

expanding the search requests; 

applying a statistical clustering algorithm, diereby grouping similar search requests together; 

identifying, using a semantic net hierai'chy, a lowest-level term in the hierarchy that subsumes 
all of the queries in a grouping of search requests; 

communicating the identified term to a user. 

25. The method of clEum 24 wherein tlie expanding is performed using a thesaums. 

26. Tlie metliod of claim 24 wherein the expanding is performed using a semantic net 
comprising synonyms and super-temis. 

27. A metliod for use witli a system storiag digital media records and comprising a search 
engine semching said stored digital media records, the method comprising the steps of: 

receiving search requests from users; 
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perfonning, by the search engine, seaiches based upon the seai'ch requests, yielding respective 
seai-ch results, each seaich result defining selected digital media records, the digital media 
records each having associated metadata; 



logging the search results and the metadata associated with digital media records selected 
therein; 

expanding the metadata, defining expanded query metadata results; 

applying a statistical clustering algorithm, thereby grouping similar expanded query metadata 
results together; 



identifying, using a semantic net hierarchy, a lowest-level temi in the hierai'chy tliat subsumes 
all of the expanded query metadata results in the grouping of expanded query metadata 
results; 

communicating the identified term to a user. 

28. The method of claim 27 wherein the expanding is performed using a thesaurus. 

29. The method of claim 27 wherein the expanding is performed using a semantic net 
comprising synonyms and super-terms. 

30. A method for use with a system storing digital media records and comprising a search 
engage seai-ching said stored digital media records, the method comprising the steps of: 

receiving first seai-ch requests fi-om a plurality of fii st users; 

performing, by the search engine, searches based upon tlie first search requests, yielding 
respective first search results, each first seaich result defining first selected digital media 
records; 
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logging the first search results; 

receiving, from the fii'st users, information indicative of subsequent actions by the fii'st users 
selecting particxilai' ones of the selected digital media records; 

5 

receiving a second search request from a second user; . 

perforaiing, by the search engine, a search based upon the second search request, yielding 
respective second search results, the second search results defining second selected digital 
10 media records; 

ranking the second selected digital media records downward for at least one second selected 
digital media record that was selected by at least one first user. 

15 31. A method for use with a system storing digital media records and comprising a search 
engine searcliing said stored digital media records, the method comprising the steps of: 

receiving first search requests from a plurality of first users; 

20 performing, by the search engine, searches based upon the first search requests, yielding 
respective first search results, each fii'st search result defining first selected digital media 
records; 

loggiag the first search results; 

25 

receiving, fi'om the first users, information indicative of subsequent actions by the first users 
selecting par ticular ones of the selected digital media records; 

receiving a second search request fi'om a second user; 

30 

performing, by the sear ch engiae, a seaxch based upon the second search request, yielding 
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respective second seaicli results, the second seaich results defining second selected digital 
media records; 

ranking the second selected digital media records downward for at least one second selected 
digital media record that was selected by at least a predetermined number of first users. 

32. A method for use with a system storing digital media records and comprising a seai'ch 
engiiie searching said stored digital media records, the method comprisiag the steps of: 

receiving first seaich requests from a plurality of first users; 

peiforaiing, by the sem'ch engine, searches based upon the first search requests, yielding 
respective fii st search results, each first search result defining first selected digital media 
records; 

logging the first search results; 

receiving, from the first users, iufonmtion indicative of subsequent actions by the first users 
selecting pai ticular ones of the selected digital media records; 

receiving a second search request from a second user; 

performing, by the search engine, a search based upon the second search request, yielding 
respective second search results, the.second search results defining second selected digital 
media records; 

ranking the second selected digital media records downward for at least one second selected 
digital media record that was selected at least a predetermined number of times by first users. 

33. A method for use with a system storing digital media records and conprising a seaich 
engine searching said stored digital media records, the method comprising the steps of: 
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receiving first search requests from a plurality of first users; 

perfoiTiiing, by the seai'cli engine, searches based upon the first seaich requests, yielding 
respective first search results, each first seai^ch result defining first selected digital media 
records; 

displaying said first selected digital media records presented in an order, said order defining 
first-presented first selected digital media record; 

logging the event of a first user purchasing said first-presented first selected digital media 
record; 

reporting, to a second user, the frequency of occurrence of the logged events. 

34. A method for use with a system storiag digital media records for puichase and 
comprising a search engiae searching said stored digital media records, the method 
comprising the steps of: 

receiving first seaich requests from a plurality of first users; 

peiforming, by the search engine, searches based upon the first search requests, yielding 
respective first search results, each fiist seaich result defining first selected digital media 
records; 

displaying said fii st selected digital media records presented in an order, said order defining 
first-presented first selected digital media record; 

logging the event of a first user issuing several consecutive search requests, and receiving 
respective first selected digital media requests greater in nmnber than a predeteimined 
threshold for said several consecutive search requests; 
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reporting, to a second user, the frequency of occunence of the logged 



events. 



35. The xBethod of cMn 34 wherein the logged events forther comprise the first user failing 
pmchase any of the stored digital media records. 

36. A niethod for use with a system storing digital media records and comprising a search 
engme searcl^ig said stored digital media records, the method comprising the steps of: 

receiving search requests from users; 

perfontiing, by the sem-ch engine, searches based upon the sem-ch requests, yielding respective 
search results, each sear-ch result defining selected digital media records or being empty; 

logging the searchrequests for which the searchresult is empty; 
expanding the logged search requests; 

applying a statistical clustering algoritl^ to the logged search requests, tiaereby grouping 
similar seai'ch requests together; 

communicating a group of search requests to a user. 

37. The method of claim 36 wherein the expanding is perfomied using a thesaurus. 

38. The method of claim 36 wherein the expanding is performed using a semantic net 
comprising synonyms and super-teims. 

39. A method for use with a system storing digital media records and comprising a search 
engme searcl^g said stored digital media records, the system pemiitting user expressions of 
mterest m particulai- stored digital media records, the method comprising the steps of: 
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performing, by the seai'ch engine, seai'ches based upon the search requests, yielding respective 
search results, each seaich result defining selected digital media records; 

logging tlie seai'ch requests for winch a user has expressed interest in a selected digital media 
record; 

expanding the logged seai'ch requests; 

applying a statistical clustering algoritlim to the logged search requests, thereby grouping 
similar search requests together; 

communicating a group of search requests to a user. 

40. Hie mediod of claim 39 wherein the expanding is perforaied using a thesaurus. 

41. The method of claim 39 wherein the expanding is performed using a semantic net 
comprising synonyms and super-terms. 

42. The method of claim 39 wherein a user expressing interest in a selected digital media 
record corcprises the user placing the selected digital media record into an online shopping 
cait. 

43. The method of claim 39 wherein a user expressing interest in a selected digital media 
record comprises the user purchasing rights to use the selected digital media record. 

44. The method of claim 39 wherein a user expressing interest in a selected digital media 
record comprises the user placing the selected digital media record into an online projects 
folder or other work space. 
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45. A method for use with a system storing digital media records and compiising a search 
engine searching said stored digital media records, the method conprising die steps of: 

receiving first search requests from a pluraUty of first users; 

perfoiming. by the seai-ch engine, searches based upon the fir st search requests, yielding 
respective first search results, each fii-st search result defining first selected digital media 
records; 

logging the fust seai-ch results; 

receiving, from the first users, infoimation indicative of subsequent actions by ihe first users 
expressing interest in particular ones of the selected digital media records; 

receiving a second search request from a second user; 

perfoiming, by the search engine, a search based upon the second search request, yielding 
respective second search results, the second search results defining second seated digital 
media records; 

ranking the second selected digital media records upward for at least one second selected 
digital media record that was for wMch at least one first user expressed interest 

46. The method of claim 45 wherein a user expressing interest in a selected digital media 
record comprises the user placing the selected digital media record into an online shopping 



cart. 



47. The method of claim 45 wherein a user expressing interest in a selected digital media 
record comprises the user purchasing rights to use the selected digital media record. 

48. The method of claim45 wherein a user expressing interest in a selected digital media 
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record comprises the user placing the selected digital media record into an online projects 
folder or other work space. 

5 49. A method for use with a system storiQg digital media records and comprising a seai'ch 
engine searching said stored digital media records, the method comprising tlie steps of: 

receiving first search requests from a plurality of first users; 

10 peifoiming, hy the search engine, seai-ches based upon the tkst search requests, yielding 
respective first search results, each first search result defining first selected digital media 
records; 

logging the fiist search results; 

15 

receiving, from the first users, information indicative of subsequent actions by the first users 
expressing interest in particular ones of the selected digital media records; 

receiving a second seai'cli request from a second user; 

20 

performing, by the search engine, a search based upon the second seaich request, yielding 
respective second search results, the second search results defining second selected digital 
media records; 

25 ranking the second selected digital media records upward for at least one second selected 
digital media record that was for which at least a predetermined number of first users 
expressed interest. 

50. The method of claim 49 wherein a user expressing interest in a selected digital media 
30 record comprises the user placing the selected digital media record into an online shopping 
cart. 
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51. TIae xnethod of claim 49 wherein a user expressing interest in a selected digital media 
record comprises the user purchasing lights to use tlae selected digital media record. 

52. Thei^tlaodofclaim49whereinauserexpiessinginterestinaselecteddigitalmedia 
record comprises the user placing the selected digital media record into an online projects 
folder or oflier work space. 

53. A method for xose witli a system storing digital media records and comprising a search 
engine searching said stored digital media records, the method comprising the steps of: 

receiving first search requests from a plurality of first users; 

perfommg, by ±e search engine, searches based upon the first search requests, yielding 
respective first search results, each first seaich result defining first selected digital media 
records; 

logging the first search results; 

receiving, from the fii st users, information indicative of subsequent actions by the first users 
expressing interest in pai ticular ones.of the selected digital media records; 

receiving a second search request from a second user; 

performing, by the search engine, a seai-chbased upon the second search request, yielding 
reepective second search results, the second search results defining second selected digital 
media records; 



ranking the second selected digital media records upward for at least one second selected 
digital media record for which first users expressed interest at least a predetermined number 



of times. 
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54. The method of claim 53 wherein a user expressing interest in a selected digital media 
record comprises the user placing tlae selected digital media record into an online shopping 
cait. 

55. The method of claim 53 wherein a user expressing interest in a selected digital media 
record comprises the user purchasing rights to use the selected digital media record. 

56. The method of claim 53 wherein a user expressing interest in a selected digital media 
record comprises the user placing the selected digital media record into an online projects 
folder or other work space. 

57. A method for use with a system storing digital media records and comprising a search 
engine searching said stored digital media records, the method comprising the steps of: 

receiving first seai'ch requests from a plurality of first users; 

performing, by the search engine, seaiches based upon the Gi st search requests, yielding 
respective first search results, each first search result defining first selected digital media 
records; 

logging the first seai'cli results; 

receiving, fioiii the first users, information indicative of subsequent actions by the first users 
expressing interest in piarticulai* ones of the selected digital media records; 

receiving a second search request from a second xiser; 

performing, by the search engine, a search based upon the second search request, yielding 
respective second search results, the second search results defining second selected digital 
media records; 
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ranking .the second selected digital media records upward for at least one second selected 
digital media record for which expression of interest satisfies a predetennined threshold, 
wherein expression of interest comprises a weighted Action of two or more of the foulwing 
events: the user placing the selected digital media record into an online shopping caxt. the 
user purchasing rights to use the selected digital media record, and the user placing thl 
selected digital media record into an online projects folder or other work space. 

58. A method for use with a metadata stream with respect to a video stream, the metadata- 
stream time-coded with respect to the video stream, the method comprising the steps, 
perfoi-med by a computer, of: 

searching tor proper names within the metadata stream; 
finding faces within the video stream; 
niatching faces with proper names; and 

. placing the matched faces and proper names into a reference Kbrary. 

59. The method of claim 58 wherein the video stream is news footage, wherein the metadata 
stieam is closed captioning for the hearing impaired. 

60. A method for use with a metadata stream with respect to at least one video stream, the 
metadata stream time-coded with respect to the at least one video stream, the metlaod 
compris; :g the steps, performed by a computer, of: 

searching for proper names within the metadata stream; 

finding faces within the at least one video stream; 

matching faces with proper names; 
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placing the matched faces and proper names into a reference library; and 

for a selected proper name, generating a list of locations withia the at least one video stream 
associated with that selected proper name. 

61. The method of claim 60 wherein the video stream is news footage, wherein the metadata 
stream is closed captioning for tire heaiing impaired. 

62. The method of claim 60 fmther comprising the step of commimicating the list of 
locations to a rights management system. 

63. A method for use with a system for managing digital media jBles, the method comprising 
tlie steps of: 

using face recognition to recognize faces portrayed in tlie digital media files, yielding 
metadata with respect to the digital media files indicative of the recognized faces; 

analyzing the metadata indicative of the recognized faces to detect duplicate files among the 
digital media files; and 

displaying the duphcate files for a user. 

64. A method for use with a system for naanaging digital media files, the method comprising 
Uie steps of: 

using image recognition to recognize images portrayed in the digital media files, yielding 
metadata witla respect to the digital media fUes indicative of the recognized images; 

analyzing the metadata indicative of the recognized images to detect duplicate files among tlie 
digital media files; and 
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displaying the duplicate files for a user. 

65. A method for use with a system for managing digital media files, the method comprising 
the steps of: 

using scene detection to detect scenes portrayed in tlie digital media files, yielding metadata 
with respect to the digital media files indicative of the detected scenes; 

analyzing the metadata indicative of the detecting scenes to detect duplicate files among the 
digital noedia files; and 

displaying the duphcate files for a user. 
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Box I Observations where certain claims were found unsearchable (Continuation of Item 1 of first sheet) 



Tills international report has not been established in respect of certain claims under A r dele 17(2)(a) for the following reasons: 

1. Q Claim Nos.: 

because they relate to subject matter not required to be searched by this Authority, namely: 



1 I Claim Nos.: 

because they relate to parts of the international application that do not conply with the prescribed requirements to 
such an extent that no meaningful mtemadonal search can be carried out, specifically: 



6.4<a). 



Claim Nos.: 

because they are dependent claims and are not drafted in accordance widi die second and third sentences of Rule 



Box n Observations where unity of invention is lacking (Continuation of Item 2 of first sheet) 



This hitemational Searching Authority found multiple invendons in this international application, as follows: 
Please See Continuation Sheet 



1X1 As all required additional search fees were timely paid by the applicant, diis intemadonal search report covers all 
^^^^ searchable claims. 

2. I I As all searchable claims could be searched without effort justify ing an additional fee» this Authority did not invite 
payment of any additional fee. 

I I As only some of the requhed additional search fees wsre timely paid by the applicant, this intematiozial search 
report covers only those claims for which fees were paid, specifically claims Nos.: 



4. I I No required addidonal search fees were timely paid by the applicant. Consequently, this intemadonal search report 
is restricted to the invention Vixst mentioned in the claims; it is covered by claims Nos.: 

Remark on Protest | | The additional search fees were accon^anied by the applicant's protest. 

I I No protest accoiiqianied the payment of additional search fees. 
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BOX n. OBSERVATIONS WHERE UNITY OF INVENTION IS LACKING 

This application contains the foUowing inventions or groups of invenUons which are not so linked as to form a sinele eeneral 
mvenuveconcept under per RtUe 13.1. In order for all inventions to be exannned. the app^J^fa^aS^^^^ 

Group I. claim{s) 1-2 and 58-62, dra%vn to use with metadata streams with respect to a video strcsam. 
Group II. claim(s) 3-8 and 63-65. drawn to thumbnail images in storing and managing digital media files. 
Group in, claim(s) 9-15, drawn to vocabulary file of words keyed to the digital media records. 
Groiqj IV, claim(s) 16-17. drawn to right management. 

Group V. claim{s) 18-23. 30-35. and 45-57. dra\vn to search engine searchtog digital media records. 

Group VI. claim(s) 24-29 and 36-44. dravro to a statistical clustering algoriflm in searching digital media records. 

IiL^r2'*rv Sfr« ^r*^' I-VI do not relate to a single general inventive concept under PCT Rule 13. 1 because, under PCT 
Rule 13.2. they lack the same or conespondmg special technical features for die foUowing reasons: the special technical feaim^of 
StSlLrg""'^''"^"- ^'-«''^'^«"^'-'>^«^f-•-°f-ho«.ofGroups^snotpr;i:^^^^^^ 



Form PCT/ISA/2I0 (second sheet) (July 1998) 



